DeepFakes Detection Based on Heart Rate Estimation: Single- and Multi-frame

This chapter describes a DeepFake detection framework based on physiological measurement. In particular, we consider information related to the heart rate using remote photoplethysmography (rPPG). rPPG methods analyze video sequences looking for subtle color changes in the human skin, revealing the presence of human blood under the tissues. This chapter explores to what extent rPPG is useful for the detection of DeepFake videos. We analyze the recent fake detector named DeepFakesON-Phys that is based on a Convolutional Attention Network (CAN), which extracts spatial and temporal information from video frames, analyzing and combining both sources to better detect fake videos. DeepFakesON-Phys has been experimentally evaluated using the latest public databases in the field: Celeb-DF v2 and DFDC. The results achieved for DeepFake detection based on a single frame are over 98% AUC (Area Under the Curve) on both databases, proving the success of fake detectors based on physiological measurement to detect the latest DeepFake videos. In this chapter, we also propose and study heuristical and statistical approaches for performing continuous DeepFake detection by combining scores from consecutive frames with low latency and high accuracy (100% on the Celeb-DF v2 evaluation dataset). We show that combining scores extracted from short-time video sequences can improve the discrimination power of DeepFakesON-PhysThis work has been supported by projects: PRIMA (H2020-MSCA-ITN2019-860315), TRESPASS-ETN (H2020-MSCA-ITN-2019-860813), BIBECA (MINECO/FEDER RTI2018-101248-B-I00), and COST CA16101 (MULTI-FORESEE). J. H.-O. is supported by a PhD fellowship from UA

A Survey of Super-Resolution in Iris Biometrics With Evaluation of Dictionary-Learning

© 2018 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other worksThe lack of resolution has a negative impact on the performance of image-based biometrics. While many generic super-resolution methods have been proposed to restore low-resolution images, they usually aim to enhance their visual appearance. However, an overall visual enhancement of biometric images does not necessarily correlate with a better recognition performance. Reconstruction approaches thus need to incorporate the specific information from the target biometric modality to effectively improve recognition performance. This paper presents a comprehensive survey of iris super-resolution approaches proposed in the literature. We have also adapted an eigen-patches’ reconstruction method based on the principal component analysis eigen-transformation of local image patches. The structure of the iris is exploited by building a patch-position-dependent dictionary. In addition, image patches are restored separately, having their own reconstruction weights. This allows the solution to be locally optimized, helping to preserve local information. To evaluate the algorithm, we degraded the high-resolution images from the CASIA Interval V3 database. Different restorations were considered, with 15 × 15 pixels being the smallest resolution evaluated. To the best of our knowledge, this is the smallest resolutions employed in the literature. The experimental framework is complemented with six publicly available iris comparators that were used to carry out biometric verification and identification experiments. The experimental results show that the proposed method significantly outperforms both the bilinear and bicubic interpolations at a very low resolution. The performance of a number of comparators attains an impressive equal error rate as low as 5% and a Top-1 accuracy of 77%–84% when considering the iris images of only 15 × 15 pixels. These results clearly demonstrate the benefit of using trained super-resolution techniques to improve the quality of iris images prior to matchingThis work was supported by the EU COST Action under Grant IC1106. The work of F. Alonso-Fernandez and J. Bigun was supported in part by the Swedish Research Council, in part by the Swedish Innovation Agency, and in part by the Swedish Knowledge Foundation through the CAISR/SIDUS-AIR projects. The work of J. Fierrez was supported by the Spanish MINECO/FEDER through the CogniMetrics Project under Grant TEC2015-70627-R. The authors acknowledge the Halmstad University Library for its support with the open access fee

GANprintR: Improved Fakes and Evaluation of the State of the Art in Face Manipulation Detection

© 2020 IEEE.  Personal use of this material is permitted.  Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other worksThe availability of large-scale facial databases, together with the remarkable progresses of deep learning technologies, in particular Generative Adversarial Networks (GANs), have led to the generation of extremely realistic fake facial content, raising obvious concerns about the potential for misuse. Such concerns have fostered the research on manipulation detection methods that, contrary to humans, have already achieved astonishing results in various scenarios. In this study, we focus on the synthesis of entire facial images, which is a specific type of facial manipulation. The main contributions of this study are four-fold: i) a novel strategy to remove GAN 'fingerprints' from synthetic fake images based on autoencoders is described, in order to spoof facial manipulation detection systems while keeping the visual quality of the resulting images; ii) an in-depth analysis of the recent literature in facial manipulation detection; iii) a complete experimental assessment of this type of facial manipulation, considering the state-of-the-art fake detection systems (based on holistic deep networks, steganalysis, and local artifacts), remarking how challenging is this task in unconstrained scenarios; and finally iv) we announce a novel public database, named iFakeFaceDB, yielding from the application of our proposed GAN-fingerprint Removal approach (GANprintR) to already very realistic synthetic fake images. The results obtained in our empirical evaluation show that additional efforts are required to develop robust facial manipulation detection systems against unseen conditions and spoof techniques, such as the one proposed in this studyThis work has been supported by projects: PRIMA (H2020-MSCA-ITN-2019-860315), TRESPASS-ETN (H2020-MSCA-ITN2019-860813), BIBECA (RTI2018-101248-B-I00 MINECO/FEDER), BioGuard (Ayudas Fundación BBVA a Equipos de Investigación Cientíifica 2017), Accenture, by NOVA LINCS (UIDB/04516/2020) with the financial support of FCT - Fundação para a Ciência e a Tecnologia, through national funds, and by FCT/MCTES through national funds and co-funded by EU under the project UIDB/EEA/50008/202

BeCAPTCHA: Behavioral bot detection using touchscreen and mobile sensors benchmarked on HuMIdb

In this paper we study the suitability of a new generation of CAPTCHA methods based on smartphone interactions. The heterogeneous flow of data generated during the interaction with the smartphones can be used to model human behavior when interacting with the technology and improve bot detection algorithms. For this, we propose BeCAPTCHA, a CAPTCHA method based on the analysis of the touchscreen information obtained during a single drag and drop task in combination with the accelerometer data. The goal of BeCAPTCHA is to determine whether the drag and drop task was realized by a human or a bot. We evaluate the method by generating fake samples synthesized with Generative Adversarial Neural Networks and handcrafted methods. Our results suggest the potential of mobile sensors to characterize the human behavior and develop a new generation of CAPTCHAs. The experiments are evaluated with HuMIdb1 (Human Mobile Interaction database), a novel multimodal mobile database that comprises 14 mobile sensors acquired from 600 users. HuMIdb is freely available to the research communityThis work has been supported by projects: PRIMA, Spain (H2020-MSCA-ITN-2019-860315), TRESPASS-ETN, Spain (H2020-MSCA-ITN-2019-860813), BIBECA RTI2018-101248-B-I00 (MINECO/FEDER), and BioGuard, Spain (Ayudas Fundación BBVA a Equipos de Investigación Científica 2017). Spanish Patent Application P20203006

Learning emotional-blinded face representations

© 2020 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other worksWe propose two face representations that are blind to facial expressions associated to emotional responses. This work is in part motivated by new international regulations for personal data protection, which enforce data controllers to protect any kind of sensitive information involved in automatic processes. The advances in Affective Computing have contributed to improve human-machine interfaces but, at the same time, the capacity to monitorize emotional responses triggers potential risks for humans, both in terms of fairness and privacy. We propose two different methods to learn these expression-blinded facial features. We show that it is possible to eliminate information related to emotion recognition tasks, while the performance of subject verification, gender recognition, and ethnicity classification are just slightly affected. We also present an application to train fairer classifiers in a case study of attractiveness classification with respect to a protected facial expression attribute. The results demonstrate that it is possible to reduce emotional information in the face representation while retaining competitive performance in other face-based artificial intelligence tasksThis work has been supported by projects: PRIMA (H2020-MSCA-ITN-2019-860315), TRESPASS-ETN (H2020-MSCAITN-2019-860813), IDEA-FAST (IMI2-2018-15-853981), BIBECA (RTI2018-101248-B-I00 MINECO/FEDER), REAVIPERO (RED2018-102511-T), RTI2018-095232-B-C22 MINECO, and Accenture. A. Pena is supported by a research fellowship (PEJ2018-004094A) from the Spanish MINEC

A Comparative Evaluation of Heart Rate Estimation Methods using Face Videos

© 2020 IEEE.  Personal use of this material is permitted.  Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other worksThis paper presents a comparative evaluation of methods for remote heart rate estimation using face videos, i.e., given a video sequence of the face as input, methods to process it to obtain a robust estimation of the subject's heart rate at each moment. Four alternatives from the literature are tested, three based in hand-crafted approaches and one based on deep learning. The methods are compared using RGB videos from the COHFACE database. Experiments show that the learning-based method achieves much better accuracy than the hand-crafted ones. The low error rate achieved by the learning-based model makes possible its application in real scenarios, e.g. in medical or sports environmentsThis work has been supported by projects: IDEAFAST (IMI2-2018-15-two-stage-853981), PRIMA (ITN-2019-860315), TRESPASS-ETN (ITN-2019-860813), BIBECA (RTI2018- 101248-B-I00 MINECO/FEDER), and edBB (UAM). J.H.O. is supported by a PhD Scholarship from UAM. Portions of the research in this paper used the COHFACE Dataset made available by the Idiap Research Institute, Martigny, Switzerlan

FairCVtest Demo: Understanding Bias in Multimodal Learning with a Testbed in Fair Automatic Recruitment

© ACM 2020. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in ICMI '20: Proceedings of the 2020 International Conference on Multimodal Interaction, https://doi.org/10.1145/3382507.3421165With the aim of studying how current multimodal AI algorithms based on heterogeneous sources of information are affected by sensitive elements and inner biases in the data, this demonstrator experiments over an automated recruitment testbed based on Curriculum Vitae: FairCVtest. The presence of decision-making algorithms in society is rapidly increasing nowadays, while concerns about their transparency and the possibility of these algorithms becoming new sources of discrimination are arising. This demo shows the capacity of the Artificial Intelligence (AI) behind a recruitment tool to extract sensitive information from unstructured data, and exploit it in combination to data biases in undesirable (unfair) ways. Aditionally, the demo includes a new algorithm (SensitiveNets) for discrimination-aware learning which eliminates sensitive information in our multimodal AI frameworkThis work has been supported by projects: BIBECA (RTI2018-101248-B-I00 from MINECO/FEDER), TRESPASS-ETN (H2020-MSCA-ITN-2019-860813), PRIMA (H2020-MSCA-ITN-2019-860315); and by Accentur

Sensitive loss: Improving accuracy and fairness of face representations with discrimination-aware deep learning

We propose a discrimination-aware learning method to improve both the accuracy and fairness of biased face recognition algorithms. The most popular face recognition benchmarks assume a distribution of subjects without paying much attention to their demographic attributes. In this work, we perform a comprehensive discrimination-aware experimentation of deep learning-based face recognition. We also propose a notational framework for algorithmic discrimination with application to face biometrics. The experiments include three popular face recognition models and three public databases composed of 64,000 identities from different demographic groups characterized by sex and ethnicity. We experimentally show that learning processes based on the most used face databases have led to popular pre-trained deep face models that present evidence of strong algorithmic discrimination. Finally, we propose a discrimination-aware learning method, Sensitive Loss, based on the popular triplet loss function and a sensitive triplet generator. Our approach works as an add-on to pre-trained networks and is used to improve their performance in terms of average accuracy and fairness. The method shows results comparable to state-of-the-art de-biasing networks and represents a step forward to prevent discriminatory automatic systemsThe authors would like to thank Manuel Cebrian and Iyad Rahwan for their constructive feedback and inspiring talks. This work has been supported by projects: TRESPASS-ETN (MSCA-ITN-2019-860813), PRIMA (MSCA-ITN-2019-860315), BIBECA (RTI2018-101248-B-I00 MINECO/FEDER), and BBforTAI (PID2021-127641OB-I00 MICINN/FEDER). I. Serna is supported by a research fellowship from the Universidad Autónoma de Madrid (FPI-UAM-2020). A. Morales is supported by the Madrid Government (Comunidad de Madrid-Spain) under the Multiannual Agreement with Autonomous University of Madrid in the line Encouragement of the Research of Young Researchers, in the context of the V PRICIT (Regional Programme of Research and Technological Innovation

TypeNet: Scaling up keystroke biometrics

© 2020 IEEE.  Personal use of this material is permitted.  Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other worksWe study the suitability of keystroke dynamics to authenticate 100 K users typing free-text. For this, we first analyze to what extent our method based on a Siamese Recurrent Neural Network (RNN) is able to authenticate users when the amount of data per user is scarce, a common scenario in free-text keystroke authentication. With 1 K users for testing the network, a population size comparable to previous works, TypeNet obtains an equal error rate of 4.8% using only 5 enrollment sequences and 1 test sequence per user with 50 keystrokes per sequence. Using the same amount of data per user, as the number of test users is scaled up to 100K, the performance in comparison to 1 K decays relatively by less than 5%, demonstrating the potential of Type-Net to scale well at large scale number of users. Our experiments are conducted with the Aalto University keystroke database. To the best of our knowledge, this is the largest free-text keystroke database captured with more than 136M keystrokes from 168K usersThis work has been supported by projects: PRIMA (MSCA-ITN-2019-860315), TRESPASS (MSCAITN-2019-860813), BIBECA (RTI2018-101248-B-I00 MINECO), and by edBB (UAM). A. Acien is supported by a FPI fellowship from the Spanish MINEC

Deepfakes and beyond: A Survey of face manipulation and fake detection

The free access to large-scale public databases, together with the fast progress of deep learning techniques, in particular Generative Adversarial Networks, have led to the generation of very realistic fake content with its corresponding implications towards society in this era of fake news. This survey provides a thorough review of techniques for manipulating face images including DeepFake methods, and methods to detect such manipulations. In particular, four types of facial manipulation are reviewed: i) entire face synthesis, ii) identity swap (DeepFakes), iii) attribute manipulation, and iv) expression swap. For each manipulation group, we provide details regarding manipulation techniques, existing public databases, and key benchmarks for technology evaluation of fake detection methods, including a summary of results from those evaluations. Among all the aspects discussed in the survey, we pay special attention to the latest generation of DeepFakes, highlighting its improvements and challenges for fake detection. In addition to the survey information, we also discuss open issues and future trends that should be considered to advance in the fieldThis work has been supported by projects: PRIMA (H2020-MSCAITN-2019-860315), TRESPASS-ETN (H2020-MSCA-ITN-2019-860813), BIBECA (MINECO/FEDER RTI2018-101248-B-I00), Bio-Guard (Ayudas Fundación BBVA a Equipos de Investigación Científica 2017), and Accenture. Ruben Tolosana is supported by Consejería de Educación, Juventud y Deporte de la Comunidad de Madrid y Fondo Social Europe